NVIDIA’s Neuralangelo AI model can create 3D objects from 2D videos

NVIDIA has introduced a new AI model called Neuralangelo that can create 3D replicas of objects from 2D videos, whether they are classical sculptures or ordinary trucks and buildings. Neuralangelo works by selecting multiple frames that show an object from different angles in a 2D video to get a clear picture of its depth, size, and shape. It then creates a rough 3D representation of the object before optimizing it to mimic the details of the real thing.

According to the company, the new model has adopted technology from its old Instant NeRF version to be able to capture the fine details of anything the user wants to recreate in 3D. These include textures, patterns and color variations. NVIDIA says Neuralangelo’s ability to capture difficult textures, such as the roughness of ceiling panels and the smoothness of marble, “significantly outperforms previous methods.” Of course, it’s a good idea to create usable virtual objects from videos taken with smartphones, which can provide creators with an easy way to create 3D games for their projects. It’s also a quick fix: NVIDIA Research told us the tool can create 3D scenes in a matter of hours.

Ming-Yu Liu, co-author of the Neuralangelo paper, said it will “ultimately enable developers to bring detailed objects — whether tiny figures or huge buildings — into virtual environments for video games or artificial digital twins.” In the video below showing off what the technology can do, NVIDIA said it can create large-scale views of drone footage, making it easier for developers to recreate real locations in their games. It has many other potential applications, including robotics, virtual reality experiences, architecture, and of course, art.

The company’s research department will present Neuralangelo at the computer vision conference and pattern recognition event, which will be held from 18 to 22. June in Vancouver.